Statistical hypothesis testing is used to improve marketing and business intelligence by comparing user responses to two or more variants of a user experience. One form of statistical hypothesis testing includes “A/B” testing in which user responses to an “A” alternative and a “B” alternative of a user experience are tested. The presence of cached content or stale experiment states on devices may result in a selection bias that favors one alternative over other alternatives due to the failure of one or more devices to obtain or output a newer experiment state. Skewed response data is not statistically valid and may obscure the presence of other errors in experimental parameters.
While implementations are described herein by way of example, those skilled in the art will recognize that the implementations are not limited to the examples or figures described. It should be understood that the figures and detailed description thereto are not intended to limit implementations to the particular form disclosed but, on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.